RMDS Server Consolidation with Solaris™ 10 Containers
White Paper June 2007
2 RMDS Server Consolidation with Solaris 10 Containers
Sun Microsystems, Inc.
Table of Contents
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Reuters Market Data System (RMDS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 A Typical Deployment Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 A Quick Introduction To Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Proposed Deployment with Solaris Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3 RMDS Server Consolidation with Solaris 10 Containers
Sun Microsystems, Inc.
Introduction
One of the most pressing challenges for Financial Services industry firms is how to improve throughput performance and reduce latency when accessing time-sensitive market data. The continued growth in the volume of market data requires an ever increasing market data infrastructure to allow the business to keep pace with these changes. At the same time customers are increasingly looking to reduce the TCO of the operational environment by exploiting standardized deployment platforms. These platforms are likely to be deployed using virtualization techniques to increase their utilization. There are many different approaches to address the needs of application scaling, ranging from vertical scaling for applications which have a highly threaded capability to horizontal scaling for those applications which perform most cost effectively on smaller systems where requirements for additional performance are addressed by additional discrete systems rather than the use of larger more powerful systems. One of the challenges of horizontal scaling is the increase in the number of servers, dramatically driving increased space, power and cooling requirements and likely resulting in a significant number of underutilized systems. Due to its inherently single threaded nature, Reuters Market Data System (RMDS) is an application that lends itself to horizontal scaling, with two processor systems typically representing the optimum environment. This however results in the challenges listed above. For many Financial Services customers space and power constraints are an increasingly limiting factor in the deployment of new systems. Additionally there is an executive focus in many companies on their environmental impact. This means alternative approaches, which allow for continued growth and increase in performance whilst addressing the needs for more effective use of power, space and improved utilization rates, are needed. It behooves systems vendors to suggest such approaches to effectively manage environmental effects of their systems, without impacting the performance curve of applications. The range of systems offered by Sun continues to grow rapidly, ranging from single socket single core systems to multi-socket multi-core systems based on SPARC® and x64 chip sets, in a variety of form-factors. The SPARC offerings include systems built using Chip Multi Threaded CPUs and the more traditional SPARC CPUs. The x64 offering are built using CPUs from our partners at AMD™ and Intel. For many years Solaris has been the preferred deployment platform for customers implementing RMDS and it's predecessor products. The release of Solaris™ 10 on both the SPARC and the x64 platforms dramatically increased Solaris' market penetration, offering customers a large number of Sun and third-party systems as deployment options. Solaris 10 also introduced a number of significant features, such as Containers, DTrace, Service Management Facility, ZFS, FireEngine, and a number of security enhancements including Trusted Solaris. Containers however are most relevant for this paper since they offer a virtualization environment that can be used to consolidate multiple small systems on a single system. This paper presents how Solaris Containers can be utilized to consolidate several RMDS servers into a fewer number of CMT or multi-core multiprocessor systems.
4 RMDS Server Consolidation with Solaris 10 Containers
Sun Microsystems, Inc.
Reuters Market Data System (RMDS)
The Reuters Market Data System (RMDS) provides an open platform for efficient and reliable distribution of Reuters, valueadded and third-party data to client applications. In response to the market demand for reduced total cost of ownership, improved performance, flexibility and ease of implementation, Reuters supports the next-generation Market Data System–RMDS6.0–on Solaris 10 on the x64 and the SPARC platforms. RDMS6.0 consists of several modules including the following three core components that will be discussed in this paper: • Source Distributor (src_dist) The Source Distributor is a key component of the Market Data Hub (MDH) . MDH is the RMDS backbone consisting of the Source Distributor, P2PS, and RTIC components that communicate using RRMP (Reuters Reliable Management Protocol). The Source Distributor is used by all source server publishers in order to make their content available on the MDH. • RTIC (Real-Time In-memory Cache) - TIC RMDS Edition The TIC–RMDS Edition (RTIC) consists of a distributed real-time, in-memory cache and various processes that perform intelligent data distribution. The cache contains the latest values of instruments provided by market data feed sources, and acts as an initial value cache for instruments it already contains. • P2PS (Point-to-Point Server) The Point-to-Point Server (P2PS) provides a consolidated point-to-point distribution solution for trading room systems. Information is distributed over proprietary Reuters protocols built on top of TCP/IP.
A Typical Deployment Scenario
Reuters' recommended systems for RMDS deployment are two socket servers, with each RMDS component on its own dedicated system. If a customer deploys RMDS in their production environment with the core modules of RMDS–Source Distributor, P2PS and RTIC–they would require at least 8 two-socket servers in a typical deployment scenario shown in Figure 1. Each of the RMDS components are deployed on their own server, These servers likely have unused capacity, however multiple instances or components are typically not consolidated especially where these components are receiving and/or sending messages on the same UDP service simultaneously. Each RMDS component requires an underlying messaging daemon which is typically TIBCO Rendezvous or Reuters Reliable Communication Protocol (RRCP). Each messaging daemon uses a UDP or PGM service, typically a combination of service name and port number. The service parameter instructs the daemon to use this tuple when talking on the transport. Multiple daemons on the same system can't use the same UDP service to send/receive messages at the same time, nor can a single daemon to send/receive messages at the same time, nor can a single daemon use the same UDP service with different multicast or broadcast addresses concurrently. These constraints make running multiple instances of an RMDS component on a single system rather complex. This in turn makes consolidation of the the RMDS infrastructure difficult and thus limits the customers ability to effectively exploit the increasing range of multi-cpu and multi-core systems to maximize performance whilst minimising operational costs.
5 RMDS Server Consolidation with Solaris 10 Containers
Sun Microsystems, Inc.
Not only does the requirement of a dedicated system for each component raise operational costs, perhaps more significantly each physical system introduces additional network latency. With the ever increasing focus on low latency trading environments largely as a result of algorithmic trading and regulations, firms are increasingly looking for innovative solutions to reduce latency. Recognizing that communications between individual systems contribute significant latency, Sun is recommending the use of Containers to eliminate network hops and thus the associated latency. The components consolidated using Containers could be RMDS components, or could be consumer applications running in Containers on the same system as the relevant RMDS components. As an example, a trading application and RMDS P2PS could be run on two Containers on the same system. The communication between P2PS and the trading application will now be at memory speeds, typically measured in nano seconds versus network latency, typically measured in micro seconds.
RMDS Standard Deployment Without Solaris Containers
Source Distributor #3
Server # 3
Source Distributor #2
Server # 2
Src_Dist RVD or RRCPD
Source Distributor #1
Server # 1
Src_Dist RVD or RRCPD
Src_Dist RVD or RRCPD
MDH Backbone # 1 RRCP or RV Transport
MDH Backbone # 2 RRCP or RV Transport
MDH Backbone # 3 RRCP or RV Transport
Server # 4
RVD or RRCPD P2PS
RTIC RVD
Server # 7
Trader
RVD or RRCPD
Server # 5
Server # 6
P2PS Server # 1 P2PS Server # 2
RTIC Server # 1 RTIC Server # 2
P2PS Server # 3
Mini Distribution Layer (Gigabit Network)
Trader
Algor Trading App Trader App
Trader
Trader
Fig 1
Server # 8
6 RMDS Server Consolidation with Solaris 10 Containers
Sun Microsystems, Inc.
A Quick Introduction To Containers
Containers were introduced with Solaris 10, and are available on all platforms that Solaris 10 is supported on. Containers allow individual instances of Solaris 10 to be “installed” and “booted” within a single global instance of Solaris 10. Each of these non-global Containers appear as individual Solaris instances, each with their own identity, user namespace, IPC namespace and network namespace. They are security and fault contained, thus one non-global Container does not have the ability to view the data or resources associated with another non-global Container. Faults inside one non-global Container cannot propagate into any other non-global Container. The administration model of Containers is simple–the administrator for the global Container has the ability to affect the entire system, including all the non-global Containers. A kernel patch applied in the global Container instantly propagates to all installed non-global zones, since the global and non-global zones share the same copy of the kernel. A non-global zone administrator however has the ability to affect only the Container they are responsible for. They can apply application level patches, and can reboot their non-global Container, however they do not have the ability to change any attribute that affects the entire system. As an example, they do not have the ability to change the system time, DTrace the kernel or reboot the system. Resources may be dedicated to Containers. These resources include CPU, memory and network bandwidth. Each Container also has the ability to run with a different default scheduling class, thus affecting runtime behavior for only the applications running within that Container. Filesystems can be dedicated to Containers, and can also be configured as shared resources between Containers. Physical devices such as tape devices and raw disk devices can be dedicated to Containers. Details of Containers may be found at: http://www.sun.com/software/solaris/Containers_learning_center.jsp and http://www.opensolaris.org/os/community/zones/ 2.4. Proposed Deployment with Solaris Containers:
Global Zone
global zone root: / web zone zone root: /zone/web app_server zone zone root: /zone/app database zone zone root: /zone/mysql
system services
(patrol)
15 web service project
(Apache 1.3.22)
60 jes project
(j2se)
70 mysql project
(mysqld)
audit services
(auditd)
10 crypto project
(ssl)
0
app users proj
(sh, bash, prstat) (inetd, sshd) zcons
20 dba users proj
(sh, bash, prstat) (inetd, sshd) zcons
security services
(login, BSM) console hme0
5
proxy project
(proxy) hme0:1 zcons ce0:1
20 system project
hme0:2
10 system project
hme0:3
ce0
ce1
10
60
default pool (1 CPU; 4GB)
zone management
(zonecfg, zoneadm, zlogin)
zoneadmd
zoneadmd
zoneadmd
pool1 (7 CPU; 3GB), FSS
core services
(inetd, rpcbind, sshd, ...)
pool2 (4 CPU; 5GB), FSS
platform administration
(syseventd, devfsadm, ifconfig,...)
remote admin/monitoring
(SNMP, SunMC, WBEM)
storage complex network device (hme0) network device (ce0) network device (ce1)
Fig A
Virtual Platform
ce0:2
/usr
ce0:3
/usr
/usr
/usr
Application Environment
7 RMDS Server Consolidation with Solaris 10 Containers
Sun Microsystems, Inc.
Proposed Deployment with Solaris Containers
Using Containers in a RMDS infrastructure would suggest deploying each RMDS component in its own Container, each Container now behaving as a “separate system”, thus allowing RMDS to scale horizontally. As market data volumes increase and individual components reach their processing limits, additional components can be simply implemented in their own Container on the same system, as long as the system has available physical resources. The use of Solaris Containers provides a range of opportunities for customers to optimize their market data infrastructure. In the example below we focus on consolidation but as discussed previously Containers provide an opportunity to improve performance by reducing latency both within the RMDS infrastructure itself and between RMDS and consumer applications. In Fig 1 we outlined a typical deployment infrastructure for an RMDS system, with each component deployed on its own system, using either Solaris on SPARC or x64, or Linux on x64. In the original architecture each of the components would be deployed on dedicated systems. The example in the diagram below (Fig 2) shows the same components installed in dedicated Containers on the same physical system. In this scenario, multiple Containers can be created on a multi-CPU or multi-core system. Different RMDS modules can be installed into these multiple virtualized environments on a single system, allowing multiple instances of RMDS modules to be run simultaneously. Each instance of an RMDS module can now process the incoming market data independently, exactly the same way it would on its own dedicated system.
RMDS Server Consolidation With Solaris Containers/Zones
Source Distributor #3
Zone # 3
Source Distributor #2
Zone # 2
Src_Dist RVD or RRCPD
Source Distributor #1
Zone # 1
Src_Dist RVD or RRCPD
Src_Dist RVD or RRCPD
MDH Backbone # 1 RRCP or RV Transport
MDH Backbone # 2 RRCP or RV Transport
MDH Backbone # 3 RRCP or RV Transport
Zone # 4
RTIC RVD
Zone # 7
Trader
RVD or RRCPD P2PS
RVD or RRCPD
Zone # 5
Zone # 6
P2PS Server # 1 P2PS Server # 2
RTIC Server # 1 RTIC Server # 2
P2PS Server # 3
Solaris 10 Containers/Zones Environment On Sun’s Multi-Core & Multi-Socket System
Mini Distribution Layer (Gigabit Network)
Trader
Algor Trading App Trader App
Trader
Trader
Fig 2
Zone # 8
8 RMDS Server Consolidation with Solaris 10 Containers
Sun Microsystems, Inc.
This approach offers a number of benefits, the specific implementation choices can be determined by the individual customer based on their specific mix of performance and cost requirements. A traditional RMDS implementation requires a physical system to be allocated to a given component. This can lead to significant under utilization of resources where a given component may only need a fraction of the systems that it is deployed on. The use of Solaris Containers allows customers to economically extract maximum value from their investment. If for example a specific RMDS module requires only 50% of a given system the remaining asset is wasted in a traditional deployment. With the use of Containers additional components could be deployed on that spare capacity, resulting in significantly lower acquisition and operational costs while driving up the overall system utilization. Typically a business will require resilience for systems as critical as market data. Under the traditional implementation model this further compounds the inefficient use of resources. Additionally when disaster recovery and contingency requirements are considered the capital acquisition costs of these under utilized systems and the cost of physically housing, operating and maintaining these systems becomes a major financial drain. Containers provide a powerful approach to minimize these costs. Another advantage of Solaris Containers is the flexibility that it offers a customer to optimize a deployment taking into account both utilization and performance. RMDS components will have utilization levels where their performance in terms of absolute throughput and latency are optimized. Typically this point will not correlate directly to a physical system sizing. In a traditional deployment this means customers must trade optimum performance against effective economic utilization. By using Containers a customer can deploy a component in a Container which is sized to deliver optimized performance and can then deploy other components in Containers each sized to optimize the performance of that component. This allows individual components to be deployed to their specific performance optimum at the same time as minimizing the total cost of deployment. Using Solaris Containers reduces the network latency between different RMDS modules when compared with RMDS implementations on individual systems. When using Containers Solaris determines that these modules are running on the same OS instance and uses the loopback interface for communication between these modules. This approach significantly reduces the network latency experienced by the introduction of addition network hops where modules are deployed on physically discrete systems. The latency benefits of Containers are not limited to the RMDS modules. With the increasing use of Algorithmic trading the importance of minimizing latency across an entire trading architecture is necessitated. In addition to implementing RMDS components on a system with Containers, Containers could be created for applications in the trading ecosystem on the same system to exploit the exceptionally low latency between the RMDS Containers and these applications. As the proposed approach is based on the exploitation of specific functionality in Solaris 10 and builds on the years of experience in running mission critical applications on the Solaris platform, the architectural approach would apply equally well for an implementation on the SPARC or x64 platforms The specific choice of platform would depend on the required characteristics of that deployment. An implementation on any Linux distribution would not be able to exploit the inherent isolation, integrity and security that Solaris Containers provide and to offer comparable functionality, security and predictable performance would need to be implemented in the traditional dedicated server model. In the original architecture where each component is implemented in a discrete system the typical recommended requirement for each system would be a dual-core, dual-cpu system in the x64 product line or a dual-cpu system for a SPARC implementation.
9 RMDS Server Consolidation with Solaris 10 Containers
Sun Microsystems, Inc.
RMDS Server Consolidation Configuration Diagram on Sun Multi-Core & Multi-Socket System
MDH Backbone # 1
Solaris Zone # 1
mdhnet subnet # 1 Interface # 1
Solaris Zone # 4
mdhnet subnet # 1 Interface # 1
Source Distributor # 1
Interface # 6 Data Feed Gigabit network Service Name = IDN DATA Server ID = 256
P2PS # 1
Interface # 5 Service Name = IDN DATA Server ID = 257 Client Subnet # 1 Gigabit network
Source Distributor # 2
Interface # 6 Service Name = IDN DATA2 Server ID = 258
mdhnet subnet # 2 Interface # 2
MDH Backbone # 2
Solaris Zone # 3
Solaris Zone # 5
mdhnet subnet # 2 Interface # 2
P2PS # 2
Service Name = IDN DATA2 Server ID = 259
Client Subnet # 2 Interface # 5
Interface # 6
Source Distributor # 3
Service Name = IDN DATA3 Server ID = 260
mdhnet subnet # 3 Interface # 3
MDH Backbone # 3
Data Feed Gigabit network
Solaris Zone #3
mdhnet subnet # 3 Interface # 3
Solaris Zone # 6
Client Subnet # 1 Gigabit network Interface # 5
P2PS # 3
Service Name = IDN DATA3 Server ID = 261
Solaris Zone #7
Clientnet Subnet # 1 Interface # 4
Solaris Zone #8
mdhnet subnet # 1 Interface # 1 mdhnet subnet # 3 Interface # 2
RTIC # 1
Service Name = IDN DATA Server ID = 265
RTIC # 2
Service Name = IDN DATA3 Server ID = 267
Clientnet Subnet # 2 Interface # 4
Single Sun Server Running 8 x Solaris Containers/Zones
Fig 3 The example above proposes the use of a SunFire™ x4600, an 8 socket AMD system available today with dual-core 2.8GHz CPUs. With this example deployment, the potential data center savings are tabulated below in Table 2. Table 1 shows the typical physical characteristics of the SunFire x4100, a 2 socket AMD system, and the SunFire x4600 systems. Table 3 shows the performance characteristics for each scenario on a per rack unit and per watt metric.
CPU Sockets X4100 X4600 2 4 RUs Typical deployment Container deployment 16 4 Total cores 4 16 Rack Units 2 4 Power 2872 1161 Typical power draw 359 1161 Onboard network Total Power of ports System (BTU/HR) 4 4 Network drops required 16 4+2=6 1396 5296 Total Power of System (BTU)/HR) 11168 5296
10 RMDS Server Consolidation with Solaris 10 Containers
Sun Microsystems, Inc.
In the following example we have considered the RMDS deployment scenario shown in Fig-1 that can, assume, handle an aggregated throughput performance of 1,000,000 (1Million) messages/sec with 8 x4100 systems. Consolidating the 8 x4100 servers into a single x4600 to handle the same throughput can provide more than 10 times better performance per Watt and 4 times better performance per Rack Unit.
Hypothetical Performance Aggregated (messages/sec) Typical deployment Container deployment 1000000 1000000 62500 250000 Performance/RU Performance/Watt (messages/watt) 21.76 215.33
Conclusion
As market data volumes continue to grow at ever increasing rates and the demand for reduced latency becomes more acute, the combination of Sun's infrastructure solutions coupled with Reuters RMDS 6 offer customers a range of new and innovative approaches to address the specific demands of their market data environment. Starting with the availability of RMDS 6 on Solaris 10 on both SPARC and x64 platforms, coupled with the announcement of the multi-CPU and multi-core systems in both the SPARC and x64 ranges, the flexibility to implement RMDS in an manner which is optimized to specific requirements has increased markedly. With the availability of support for Containers in RMDS 6 and Sun's continued innovation across the platform offerings, Sun and Reuters have never been better placed to work to deliver an optimized environment for market data which can simultaneously address performance, price and overall economic efficiency while ensuring the integrity, security and predictability of a market data environment.
RMDS Server Consolidation with Solaris 10 Containers
sun.com
© 2007 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, CA 95054 USA All rights reserved. This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any. Third-party software, including font technology, is copyrighted and licensed from Sun suppliers. Sun, Sun Microsystems, the Sun logo, Solaris, SunFire are trademarks, registered trademarks, or service marks of Sun Microsystems, Inc. in the U.S. and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. AMD, Opteron, the AMD logo, the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. RESTRICTED RIGHTS: Use, duplication, or disclosure by the U.S. Government is subject to restrictions of FAR 52.227-14(g)(2)(6/87) and FAR 52.227-19(6/87), or DFAR 252.227-7015(b)(6/95) and DFAR 227.7202-3(a). DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS HELD TO BE LEGALLY INVALID.
Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN Web sun.com
©2007 Sun Microsystems, Inc. All right reserved Sun, Sun Microsystems, the Sun logo, Solaris, SunFire are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries."